Add optional repository metadata columns (teams, topics, custom properties) #78

Copilot · 2025-06-13T21:30:05Z

This PR implements support for including extended repository metadata in CSV exports, addressing the feature request for adding repository teams, topics, and custom properties as additional columns.

Changes Made

New Environment Variable

Added INCLUDE_REPO_METADATA environment variable (default: false)
When set to true, enables fetching and including extended repository metadata
Gated behind a flag to prevent performance impact for default users

API Enhancements

Added get_repo_metadata() function to fetch repository teams, topics, and custom properties
Added make_single_api_call() helper for non-paginated API requests
Comprehensive error handling with warnings for failed metadata calls

CSV Column Extensions

Extended all CSV writing functions across all modules to include new columns when enabled:

New columns added:

repo_teams: Comma-separated list of team names with repository access
repo_topics: Comma-separated list of repository topics
repo_custom_properties: JSON string of custom repository properties

Modules updated:

code_scanning.py: All write functions (repo, org, enterprise server, enterprise cloud)
secret_scanning.py: All write functions (repo, org, enterprise)
dependabot.py: All write functions (repo, org/enterprise)

Documentation

Updated README.md with new environment variable documentation
Added usage examples showing how to enable the feature
Included performance warnings about increased API usage

Usage Example

- name: CSV export with extended metadata
  uses: advanced-security/ghas-to-csv@v3
  env:
    GITHUB_PAT: ${{ secrets.PAT }}
    GITHUB_REPORT_SCOPE: "organization"
    SCOPE_NAME: "org-name-goes-here"
    INCLUDE_REPO_METADATA: "true"

Performance Considerations

⚠️ Warning: Enabling this feature will make additional API calls for each unique repository in the results:

/repos/{owner}/{repo}/teams - for repository teams
/repos/{owner}/{repo} - for topics (included in repository details)
/repos/{owner}/{repo}/properties - for custom properties

This can significantly increase execution time and API usage when used at organization or enterprise scope, which is why it's gated behind an opt-in flag.

Backward Compatibility

All existing functionality remains unchanged when the flag is disabled (default behavior)
No breaking changes to existing CSV formats or function signatures
Feature gracefully degrades if metadata cannot be fetched (empty values provided)

Fixes #61.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

…epo metadata Co-authored-by: felickz <[email protected]>

src/api_helpers.py

+        teams_url = f"{api_endpoint}/repos/{repo_name}/teams?per_page=100&page=1"
+        teams = make_api_call(teams_url, github_pat)
+        metadata["teams"] = [team["name"] for team in teams]
+    except Exception as e:


To address the issue, we will sanitize the log messages to avoid exposing sensitive information. Specifically:

Replace the repository name (repo_name) with a generic placeholder or omit it entirely.

Avoid logging the full exception message (e) and instead log a generic error message or a sanitized version of the exception.

We will modify the print statements in src/api_helpers.py to ensure no sensitive data is logged. This involves replacing the repository name and exception details with non-sensitive placeholders.

src/api_helpers.py

+        repo_url = f"{api_endpoint}/repos/{repo_name}"
+        repo_data = make_single_api_call(repo_url, github_pat)
+        metadata["topics"] = repo_data.get("topics", [])
+    except Exception as e:


To fix the issue, we will sanitize the logging statements to ensure that sensitive data is not exposed. Specifically:

Replace the direct logging of repo_name and e with a generic warning message that does not include sensitive details.

If additional context is needed for debugging, log only non-sensitive information or use a secure logging mechanism that restricts access to sensitive logs.

For the flagged line in src/api_helpers.py, we will modify the print statement to exclude repo_name and e. Instead, we will log a generic warning message indicating that fetching repository details failed.

src/api_helpers.py

+        properties_url = f"{api_endpoint}/repos/{repo_name}/properties"
+        properties = make_single_api_call(properties_url, github_pat)
+        metadata["custom_properties"] = properties
+    except Exception as e:


To fix the issue, we will sanitize the logging statements to avoid exposing sensitive information. Specifically:

Replace the logging of repo_name with a generic placeholder or a sanitized version.

Avoid logging the full exception details (e) and instead log a generic error message or a sanitized version of the exception.

We will modify the logging statements in src/api_helpers.py to redact sensitive information while preserving the utility of the logs for debugging purposes.

src/secret_scanning.py

+                        str(metadata["custom_properties"])
+                    ])
+                except Exception as e:
+                    print(f"Warning: Failed to get metadata for {alert['repository']['full_name']}: {e}")


To fix the problem, avoid logging sensitive values like alert['repository']['full_name'] in clear text warning/error messages. When an error occurs during metadata fetching, a generic warning can be logged instead, or the repository name can be redacted or omitted. You can log an anonymized or masked version of the repository name if any identifier is required for debugging, but totally omitting it may be safest. The remainder of the error message should remain as-is to preserve debugging utility.

Edit src/secret_scanning.py on or around line 270 as follows:

Replace the f-string that prints the repository full name and error with a generic message that omits or redacts the repo name.

You may write, for instance: print("Warning: Failed to get metadata for a repository:", e).

No new imports or method definitions are needed.

…ture Co-authored-by: felickz <[email protected]>

src/secret_scanning.py

+                        str(metadata["custom_properties"])
+                    ])
+                except Exception as e:
+                    print(f"Warning: Failed to get metadata for {alert['repository']['full_name']}: {e}")


To fix the problem, we should avoid logging sensitive or potentially sensitive information (the full repository name and possibly exception details) directly as clear text. Instead, we can provide a generic warning message indicating that metadata retrieval failed, optionally including a benign identifier (such as the repository's index in the list or a generic label), but omitting sensitive fields.

Changes needed (src/secret_scanning.py, method write_enterprise_ss_list):

On line 412, modify the print() statement so it does not include alert['repository']['full_name'] or the full exception details.

The message should still indicate that a failure occurred, which can be valuable when debugging, but without exposing sensitive information.

Implementation:

Simply replace the line with a generic warning like:
print("Warning: Failed to get repository metadata for one entry.")

No new imports or method definitions are required.

Initial plan for issue

259fb1b

Copilot AI assigned Copilot and felickz Jun 13, 2025

Copilot started work on behalf of felickz June 13, 2025 21:30 View session

Add environment variable flag and update CSV functions for extended r…

4a35d76

…epo metadata Co-authored-by: felickz <[email protected]>

github-advanced-security bot found potential problems Jun 13, 2025

View reviewed changes

Complete enterprise functions and documentation for repo metadata fea…

051f090

…ture Co-authored-by: felickz <[email protected]>

github-advanced-security bot found potential problems Jun 13, 2025

View reviewed changes

Copilot AI changed the title ~~[WIP] Feature Request - Add Repo-> Team/Topic/Custom-Properties column~~ Add optional repository metadata columns (teams, topics, custom properties) Jun 13, 2025

Copilot AI requested a review from felickz June 13, 2025 21:52

Copilot finished work on behalf of felickz June 13, 2025 21:52

Merge branch 'main' into copilot/fix-61

6c917c2

@@ -42,3 +42,3 @@
                 except Exception as e:
-                    print(f"Warning: Could not fetch teams for {repo_name}: {e}")
+                    print("Warning: Could not fetch teams for the repository. Please check the logs for more details.")
                     metadata["teams"] = []
@@ -51,3 +51,3 @@
                 except Exception as e:
-                    print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
+                    print("Warning: Could not fetch repository details. Please check the logs for more details.")
                     metadata["topics"] = []
@@ -60,3 +60,3 @@
                 except Exception as e:
-                    print(f"Warning: Could not fetch custom properties for {repo_name}: {e}")
+                    print("Warning: Could not fetch custom properties for the repository. Please check the logs for more details.")
                     metadata["custom_properties"] = {}

@@ -51,3 +51,3 @@
                 except Exception as e:
-                    print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
+                    print("Warning: Could not fetch repository details. Please check the logs for more information.")
                     metadata["topics"] = []

@@ -42,3 +42,3 @@
                 except Exception as e:
-                    print(f"Warning: Could not fetch teams for {repo_name}: {e}")
+                    print(f"Warning: Could not fetch teams for the repository. Error: {str(e).splitlines()[0]}")
                     metadata["teams"] = []
@@ -51,3 +51,3 @@
                 except Exception as e:
-                    print(f"Warning: Could not fetch repository details for {repo_name}: {e}")
+                    print(f"Warning: Could not fetch repository details. Error: {str(e).splitlines()[0]}")
                     metadata["topics"] = []
@@ -60,3 +60,3 @@
                 except Exception as e:
-                    print(f"Warning: Could not fetch custom properties for {repo_name}: {e}")
+                    print(f"Warning: Could not fetch custom properties for the repository. Error: {str(e).splitlines()[0]}")
                     metadata["custom_properties"] = {}

@@ -267,7 +267,7 @@
                                     str(metadata["custom_properties"])
                                 ])
                             except Exception as e:
-                                print(f"Warning: Failed to get metadata for {alert['repository']['full_name']}: {e}")
+                                print(f"Warning: Failed to get metadata for a repository: {e}")
                                 row_data.extend(["", "", ""])
                         elif include_repo_metadata:
                             # If metadata is requested but API details not provided

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add optional repository metadata columns (teams, topics, custom properties) #78

Add optional repository metadata columns (teams, topics, custom properties) #78

Uh oh!

Copilot AI commented Jun 13, 2025 •

edited

Loading

Uh oh!

Check failure

Copilot Autofix

Check failure

Copilot Autofix

Check failure

Copilot Autofix

Check failure

Copilot Autofix

Check failure

Copilot Autofix

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add optional repository metadata columns (teams, topics, custom properties) #78

Are you sure you want to change the base?

Add optional repository metadata columns (teams, topics, custom properties) #78

Uh oh!

Conversation

Copilot AI commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made

New Environment Variable

API Enhancements

CSV Column Extensions

Documentation

Usage Example

Performance Considerations

Backward Compatibility

Uh oh!

Check failure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot Autofix

Check failure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot Autofix

Check failure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot Autofix

Check failure

Uh oh!

Uh oh!

Uh oh!

Copilot Autofix

Check failure

Uh oh!

Uh oh!

Uh oh!

Copilot Autofix

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Jun 13, 2025 •

edited

Loading